Multisensory correlation computations in the human brain identified by a time-resolved encoding model

Neural mechanisms that arbitrate between integrating and segregating multisensory information are essential for complex scene analysis and for the resolution of the multisensory correspondence problem. However, these mechanisms and their dynamics remain largely unknown, partly because classical models of multisensory integration are static. Here, we used the Multisensory Correlation Detector, a model that provides a good explanatory power for human behavior while incorporating dynamic computations. Participants judged whether sequences of auditory and visual signals originated from the same source (causal inference) or whether one modality was leading the other (temporal order), while being recorded with magnetoencephalography. First, we confirm that the Multisensory Correlation Detector explains causal inference and temporal order behavioral judgments well. Second, we found strong fits of brain activity to the two outputs of the Multisensory Correlation Detector in temporo-parietal cortices. Finally, we report an asymmetry in the goodness of the fits, which were more reliable during the causal inference task than during the temporal order judgment task. Overall, our results suggest the existence of multisensory correlation detectors in the human brain, which explain why and how causal inference is strongly driven by the temporal correlation of multisensory signals.


Supplementary Figure 2. Unexpected differences in reaction times in causality and temporal order judgment blocks.
In a given trial, following the presentation of an audiovisual sequence, participants were asked to withhold their answers until an auditory cue prompted them to answer. The Multisensory Correlation Detector model makes no explicit predictions about reaction times. However, we observe that, although stimuli are strictly identical, reaction times differ as a function of the task. Nonetheless, considering that the response delay was 1s, it is likely not long enough to completely dissociate action selection from sensory processes. Reaction times in the A. Causality judgment blocks were systematically lower than reaction times in the B. Temporal order judgment blocks (mixed-model linear regression, p < 10 -15 ). Error bars represent 2 s.e.m. across participants (N=13).

Supplementary Figure 3. Individual behavioral data and fits by the MCD model. A.
Individual fits for the causality judgments by the MCD CORR . B. Individual fits for the temporal order judgments by the MCD LAG .
Supplementary Figure 4. Late task related evoked activity and cortical source estimates. A. Spatiotemporal-cluster analysis contrasting brain activity evoked by the presentation of identical audiovisual sequences but in two different tasks (causalitytemporal order). In the contrast of causality vs. temporal order judgment blocks, a second bilateral cluster was found from central to frontal sensors (white sensors, one sample two-sided t-test, p < 0.05 corrected for multiple comparison). The peak significance was around 1600 ms, and lasted from 1445 ms to the end of the considered epochs ([1445 -1800 ms] for the left hemisphere; [1475 -1800 ms] for the right hemisphere). During this time window, the amplitude of the signal was consistently higher in the causality judgment blocks than in the temporal order judgment blocks. The top and bottom panels represent the two polarities of a single source (positive left, negative right). B. A logistic regression on the probability of a response showed no link between activity in the cluster and behavior (all p > 0.9). C. Source reconstruction revealed that this cluster was located in the left motor and left premotor cortices as well as in bilateral Superior Parietal Gyrus (SPG). This was consistent with the side of participants' right hand response and the response prompt occurring between 1800 and 2200 ms. Due to the prompting of participants to respond and the task instructions focusing on accuracy, we did not expect differences in RTs between experimental conditions. However, we did find significantly faster responses in causality judgments than in temporal order judgments. This cluster indeed captures this difference (p < 0.05). Shaded areas and error bars represent 2 s.e.m. across participants.

Supplementary Figure 5. Control spatiotemporal-cluster analysis using the signals without
performing the removal of the unisensory-specific activity. Spatiotemporal-cluster analysis contrasting brain activity evoked by the presentation of identical audiovisual sequences but in two different tasks (causality vs. temporal order judgment). Left: t-map of the significant cluster (white sensors, one sample two-sided t-test, p < 0.05 corrected for multiple comparison) ranging from 200 to 1250 ms post-sequence onset. Grey levels are t-values averaged across significant times. Right: Temporal extent of the effect averaged over significant sensors in the cluster (grey). The top and bottom panels represent the two polarities of a single source (positive left, negative right). This analysis replicates the results presented in Fig. 3A, using the signals without performing the removal of the unisensory-specific activity. Shaded areas represent 2 s.e.m. across participants.

Supplementary Figure 6. Control spatiotemporal-cluster analysis splitted by fingers.
Spatiotemporal-cluster analysis contrasting brain activity evoked by the presentation of identical audiovisual sequences but in two different tasks (causality vs. temporal order judgment). Left: t-map of the significant cluster (white sensors, p < 0.05 corrected for multiple comparison) ranging from 260 to 1250 ms post-sequence onset. Grey levels are t-values averaged across significant times. Right: Temporal extent of the effect averaged over significant sensors in the cluster (grey). The top and bottom panels represent the two polarities of a single source (positive left, negative right). This analysis replicates the results presented in Fig. 3A. A spatiotemporal clustering analysis on a 2 by 2 ANOVA with factor Task identity (2) and Finger identity (2) showed no significant effects of Finger identity (p > 0.05 for all clusters on the F-values associated with Finger identity) or interaction between Task and Finger identity (p > 0.05 for all clusters on the F-values associated with the interaction Task identity x Finger identity). Shaded areas represent 2 s.e.m. across participants.